The present invention relates to video coding technique for mobile communication of digital media data, and in particular, to processing of received data when packet loss occurs.
As mobile communication service of the next generation, it is expected that digital media data file compressed and stored in delivery server will be received and decoded at wireless portable terminal such as portable telephone. As the service of communication when packet loss (loss of packet data in transmission) occurs in the wireless mobile communication service as described above, the following two modes are anticipated:    1) Streaming service: Unreliable transport protocol (e.g. UDP: User Datagram Protocol; RTP: Real-time Transport Protocol) is used without retransmitting loss packet (general term for unit of data or mass of data), and the data are received while receiving.    2) Download service: Reliable transport protocol to retransmit loss packet is used (e.g. TCP: Transmission Control Protocol; This is suitable for application requiring reliability because error detection and acknowledgment processing are performed to all of the messages. However, when the condition of transmission line is poor, reduction in the throughput may often occur due to frequent acknowledgment and retransmission request). All data are received and decoding processing is started after storing the data. (The details of UDP, RTP and TCP will be described later.)
Here, transport layer is a level to play a role “to deliver the data to the partner or to the other party in reliable manner” in a standard reference model of network architecture to achieve communication between different types of devices (OSI: Open Systems Interconnection; open type system interconnection; physical layer—data link layer—network layer—transport layer in the order from the position closer to communication line). IP (Internet Protocol) playing a role “to carry out address management or route selection” corresponds to the network layer. Of the above two types of services, in the download service, retransmission request is made to all packets with data loss, and transmission error is not contained in the received data. On the other hand, in the streaming service, even when there is a packet with data loss, it is not retransmitted, and the received data may contain transmission error.
Here, a concrete example is given. FIG. 2 is a diagram showing system arrangement of a portable telephone, and FIG. 3 shows packets to be transmitted. Let us assume a case where a packet string (a type of media data) of the transport layer shown in FIG. 3 is transmitted toward a portable telephone 200. Regardless of whether it is TCP or UDP-RTP, one packet of transport layer comprises header (411, 421, 431, 441, and 451) and payload (412, 422, 432, 442, and 452) of the transport layer. In this example, the payload is an actual media data. On the header, the following data are described: data amount (in byte) of the payload, serial number of the packet (sequence number), payload type (only in case of RTP; In case of TCP, data of two or more different types are not handled at one time. Identification of video data and audio data is performed on this data), and time-stamp (only in case of RTP). First, a wireless signal received from an antenna 201 is electrically converted at a wireless interface unit 202, and it is inputted to a system decoder 205 in form of a packet 203 of data link protocol. At the system decoder 205, the inputted data link protocol packet is converted to a network packet and transport packet in this order. Only the transport payload is outputted to a memory 207. At the system decoder, error detection processing in the packet and identification of payload type in each packet (only in case of RTP) are carried out. Further, when media data of different payload types are delivered at the same time, the media data are arranged for each payload type, and these are outputted to the memory 207 while being synchronized with each other.
In case the transport protocol is TCP, the system decoder sends an acknowledgment information 204 to the wireless interface unit 202 for each packet arrived without error. Each of the acknowledgment information is converted to a wireless signal at the wireless interface unit 202 and is sent back to the transmission side. At the transmission side, after waiting for the acknowledgment of the transmitted packet, the next packet is transmitted (a plurality of packets may be processed in parallel). When the acknowledgment does not arrive after a certain time has elapsed from the transmission, the packet is retransmitted to the portable telephone 200. In this respect, no packet loss occurs in TCP, and the data comprising only the payloads as shown by 2060 in FIG. 4 are outputted to the memory 207 as the payload data 206. The payload data as mentioned here is generally the data of file format, which includes images, sounds, etc. in case of download. More concretely, these are MP4 file format of MPEG-4, QuickTime of Apple, WMT (Windows Media Technology) of Microsoft.
In case the transport protocol is UDP-RTP, the system decoder provides no acknowledgment, and the packet loss is allowed. For instance, in case a second packet (421 and 422) shown in FIG. 3 does not arrive at the system decoder, or in case transmission error is included in the second packet, the data comprising only the parts of payload except the second packet as shown by 2061 in FIG. 5 are outputted to the memory 207 as the payload data 206. The payload data as mentioned here are the data where packets of video coding data and packets of audio coding data are mixed together in case of streaming. Thus, a plurality of media data (e.g. video data and audio data) are delivered at the same time. When it is necessary to decode these data at the same time, the types of media decoders of each packet (e.g. video decoder, audio decoder) must be decoded according to the payload type contained in the header of RTP packet at the system decoder 205. Also, it is necessary to output the data to the memory 207 as media data of different types.
Then, the data stored at the memory 207 are sequentially outputted to an application decoder 209 as the media data 208 at the moment when the final packet of each decoding timing has arrived in case of streaming. In the present specification, the application decoder 209 is a general term for file decoder of file format and media decoder such as video decoder, audio decoder, etc. When it is defined as an application decoder, it is assumed there is no limitation to internal structure. In the application decoder, each of the media data is decoded at proper timing while error resilience of loss packet is performed. However, error resilience contained in the media decoder is not standardized. Thus, image quality or audio quality of the reconstructed image differs according to performance characteristics of error resilience when the packet loss is contained in the media data 208.
When packet loss occurs during transmission, the media data to be delivered to the application decoder is turned to such a form that payload of the packet with loss is omitted. In general, at the application decoder, when a combination of the data string not complying with the applicable standard decoding scheme is detected in the inputted data, it is judged that a transmission error has occurred. For this reason, when the data string connecting the packets before and after the loss packet is complying with the applicable standard decoding scheme, accurate position of the transmission error cannot be detected.
Also, in order to decode the data with packet loss to an image with high quality, it is necessary that the application decoder detects the transmission error so that the influence of error may not be displayed on the image quality. Because the processing amount of the decoder is increased as the result of such error correction processing, the requirements in decoding speed or image size cannot be satisfied in an application where high-speed processor cannot be used from the reasons such as product price or power consumption.